Add HTTP resilience mode #176
Conversation
|
I have this working offline with PA and IA, both with the similar logic removed from the specific scrapers. So I think this is ready to merge. @showerst - is your team running a setup where you are comfortable keeping scrapers at an earlier version? I'm guessing yes but wanted to double check. Obviously I'll want to merge this in, but also make a follow-up merge in (Edit: forgot that FL is spatula-based ... I'm not going to patch this stuff upstream in spatula... so forget FL) It's not urgent, so if it's more comfortable for me to wait a week or something that's no problem. |
|
Yeah i'm on a fork for where we can pick and choose -- feel free to merge all the changes you need in both projects and we'll get caught up when we're ready. Thanks for the H/U. |
This is a WIP draft of porting the HTTP resilience features that @elseagle introduced (originally in the FL scraper, later I re-used in IA scraper) up into openstates-core. The goal is to turn these features into a toggleable behavior instead of something that needs to be grafted onto scrapers one-by-one.
This is not sufficiently tested yet. I'm running an IA scrape (with the grafted version removed from scraper code), and will want to do the same with FL. And then test with some other scrapers to make sure I haven't regressed something.
cc @showerst and @alexobaseki